...as in bug 1357520, but for xpcshell instead of mochitest. This bug aims to add a --verify option for the xpcshell test harness, supporting test verification. Test verification involves running a test, or small group of tests, repeatedly and possibly in different environments, with the intent of finding intermittent test failures quickly and efficiently.

Sorry, this is a big patch, but there's really not a lot going on -- just re-structuring in preparation for the next 2 patches.

runTests() has grown to almost 400 lines of code: https://dxr.mozilla.org/mozilla-central/rev/b95b1638db48fc3d450b95b98da6bcd2f9326d2f/testing/xpcshell/runxpcshelltests.py#1117-1516 This splits it up a little, with the main split being, runTests constructs a list of tests; runTestList runs the list. This makes the --verify changes easier...

I've tried to make the --verify support here very similar to that added for mochitests: Apply a list of verification steps to each test. Demonstrating no harm done: https://treeherder.mozilla.org/#/jobs?repo=try&revision=03c0849f4894af74f27504c899967859f7c5b6fb I've also completed local testing including: Desktop (Linux): mach xpcshell-test <test> mach xpcshell-test <directory> mach xpcshell-test <directory> --sequential mach xpcshell-test <test> --verify mach xpcshell-test <test1> <test2> --verify Android: mach xpcshell-test <test> mach xpcshell-test <directory> I will also demonstrate --verify mode on treeherder in bug 1380122.

Comment on attachment 8897652 [details] [diff] [review] simplify xpcshell interfaces Review of attachment 8897652 [details] [diff] [review]: ----------------------------------------------------------------- a lot of changes, but effectively at parity

Comment on attachment 8897658 [details] [diff] [review] support --verify Review of attachment 8897658 [details] [diff] [review]: ----------------------------------------------------------------- nice- I do wonder if we can end up with a common verify routine for mochitest/xpcshell- maybe after reftest is supported we could look for a smarter way to refactor this.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/e95891f5a161 1. Simplify argument passing in xpcshell test harness; r=jmaher https://hg.mozilla.org/integration/mozilla-inbound/rev/5be1bdef850a 2. Break xpcshell test harness runTests() function into smaller pieces; r=jmaher https://hg.mozilla.org/integration/mozilla-inbound/rev/11f8164a5176 3. Add --verify and --verify-max-time options to xpcshell test harness; r=jmaher

https://hg.mozilla.org/mozilla-central/rev/e95891f5a161 https://hg.mozilla.org/mozilla-central/rev/5be1bdef850a https://hg.mozilla.org/mozilla-central/rev/11f8164a5176